Zero-Shot Scene Graph Relation Prediction Through Commonsense Knowledge Integration

Relation prediction among entities in images is an important step scene graph generation (SGG), which further impacts various visual understanding and reasoning tasks. Existing SGG frameworks, however, require heavy training yet are incapable of modeling unseen (i.e., zero-shot) triplets. In this work, we stress that such incapability due to the lack commonsense reasoning, i.e., ability associate similar infer relations based on general world. To fill gap, propose CommOnsense-integrAted sCene grapH rElation pRediction (COACHER), a framework integrate knowledge for SGG, especially zero-shot relation prediction. Specifically, develop novel mining pipelines model neighborhoods paths around external graph, them top state-of-the-art frameworks. Extensive quantitative evaluations qualitative case studies both original manipulated datasets from Visual Genome demonstrate effectiveness our proposed approach. The code available at https://github.com/Wayfear/Coacher.